Transparent intelligent network for data and voice

ABSTRACT

A transparent intelligent communication network having both terrestrial and satellite links between nodes and providing improved accuracy and speed in transmission by distinguishing between data requiring rapid receipt at the receiving node and other data, and dispatching data accordingly, while transmitting system protocol and error-correcting information (e.g., retransmissions) over terrestrial links.

This is a continuation, of application Ser. No. 912,118, filed June 2,1978 now abandoned.

This invention relates to communications systems and more particularlyto communications systems providing for the digital transmission ofvoice and data.

Communications systems of the foregoing type have heretofore beenproposed, illustrative of which is that described in U.S. Pat. No.3,749,845 granted to Alexander G. Fraser on July 31, 1973 and entitled"Digital Data Communication System". Other representative examples arethose in which messages are switched through predetermined paths fromorigin to destination. In such networks, messages may be accumulated atthe point of entry into the system and then transmitted in a store andforward mode wholly or partially over the predetermined path eithersolely or in asynchronous time shared multiplex with other messages.

Another system of the prior art is represented by packet switchednetworks in which asymmetrical data is encoded into small packets whichare then sent through the system over a number of different paths at thesame time in a store and forward mode.

Still another system of the prior art involves generation of compositepackets carrying asynchronous data which are transmitted over circuitswitched paths.

As will be recognized by those skilled in the art, circuit switchingcontemplates a circuit which conducts communications between a giventerminal at one end of the system and another pre-identified terminal atthe other end.

While the foregoing systems have constituted substantial advances in thecommunications art, they nevertheless continue to be characterized bycertain disadvantages. Among these are excessive fixed and variabledelays, lack of a global flow control scheme to avoid congestion, withinthe network, limited utilization of available communications channelcapacity, out of sequence or lost packets, dependence upon userprotocol, limited error correction capability particularly onsynchronous data transmitted over circuit switched paths, inability totransmit encoded voice, intelligibly, and difficulty of using integratedterrestrial and satellite links with "roof top" antennas.

Accordingly, efforts have continued to develop improved communicationsnetworks which are free of the foregoing disadvantages.

It is one general object of this invention to improve networkcommunications having bulk data and real time traffic with permanentcircuit and switched circuit mode operation.

It is another object of this invention to simplify interconnections withexternal equipment, that is, for example either by providingtransparency in protocol or by interfacing with X-25 internationalstandard network access protocol.

It is another object of this invention to minimize delays in asubscriber achieving connection to the communications system, such beingnormally referred to as admission delays, for example by eliminating theuse of network interface machines which are necessary whennon-programmable terminals enter the system to communicate with acomputer programmed to use X-25 international standard.

It is another object of this invention to eliminate or minimize internaldelays within the network, that is, to eliminate or substantiallyminimize the steady state synchronous queues in all nodes.

It is another object of this invention to maximize utilization ofinternal system channel capacity, or bandwidth.

It is another object of this invention to provide such rapid transitthrough the system that it appears to be transparent in time; forexample, by servicing the incoming lines at rapid and regular intervalsand by trading off the satellite up-down delay with the packet formationdelay experienced in the prior art.

It is another object of this invention to provide in a time transparentcommunications network, error correction which results in highlyaccurate transmission; for example, by transferring queued portions ofthe incoming messages from the entrance to exit nodes at higher speedsthan the outgoing line speed.

It is another object of this invention to improve effective utilizationof equipment, particularly storage capacity of the entrance nodes,thereby reducing costs.

It is yet another object of this invention to improve flexibility byproviding a system which is not dependent upon user protocol orsoftware; for example, by allowing the polling and addressing ofterminals without using emulation techniques in computers using X-25international standard protocol.

It is yet one final object of the invention to provide all of theforegoing advantages in a general purpose public and/or private networkfor data and encoded as well as scrambled voice, thereby reducing thecost and increasing security of long distance telephone calls.

Accordingly, in accordance with one feature of the invention, systemarchitecture is characterized by being of the 2-level hierarchial type,as for example, by a plurality of near neighbour connected star netseach having a central node, and wherein said central nodes are eachdirectly connected to each other via 12-14 GHz satellite channels aswell as terrestrial links which carry mostly protocol messages andre-transmissions for the purposes of error correction, although somepart or all of the real time traffic can also be sent over theterrestrial links, thereby providing no more than three transmissionhops between point of entry and destination, and contributing to speedand accuracy of transmission.

In accordance with another feature of the invention, a unique entryarrangement is provided whereby priority is first accorded voicetransmission and where remaining synchronous traffic is served on aselective first in-first out basis, thus contributing to reduction inadmission delays, as well as improved transmission efficiency.

In accordance with still another feature of the invention, voice andother synchronous traffic is accorded priority over asynchronoustraffic, thereby further contributing to efficiency and effectiveness ofthe network.

In accordance with yet another feature of the invention, when servicingcomputers using X-25 standard, packets which arrive in batch form overhigh speed lines at the system are repetitively sampled at extremelybrief intervals and transmitted using idle servers and extensionmini-packets, thus contributing to the aforementioned reduction inadmission delays.

In accordance with yet another feature of the invention, information inthe form of mini-packets is rapidly extracted from input buffers tolimit the buildup of incoming information (which is normally storeduntil a complete message or packet is received in the prior art), andsuch mini-packets are grouped in multi-user packets at very briefintervals and high repetition rates, thus resulting in a parallel typeof transmission which further reduces internal delays and minimizesqueues.

In accordance with yet another feature of the invention, the multi-userpackets are transmitted rapidly from one node to another where themulti-user packets are disassembled and reassembled into othermulti-user packets for immediate progressive dispatch through succeedingportions of the network.

In accordance with still another feature of the invention through theadvantageous selection of multi-user packet contents, block checking maybe advantageously employed and multi-user packets which are receivedwith erroneous information are promptly retransmitted, thereby markedlyenhancing reliability and accuracy of transmission.

In accordance with still another feature of the invention, circuitry isadvantageously provided which recognizes when channel capacity is aboutto be under-utilized and, consequently, allocates available channelcapacity to increase the effective rate with which other messages aretransmitted through the system, thus not only contributing to reductionof transmission time through the network but, additionally, increasingthe speed with which other traffic can obtain admission to the system.

In accordance with yet another feature of the invention, where trafficsuch as encoded voice is being entered into the system, "reservations"are automatically made for the subsequent channels within the system soas to provide continuing priority and rapidity of transmission in orderthat end-to-end message delay variance may be avoided.

In accordance with still one further feature of the invention,provisions are made for integrated terrestrial and satellitecommunications links between nodes, thereby providing not only increasedflexibility and reliability for real time traffic but in addition,capacity for economic transmission of bulk data by the use of 12-14 GHzsatellite channels using "roof top" antennas.

These and other objects and features of the invention will be evidentfrom the following detailed description by way of example, withreference to the drawing in which:

FIG. 1 depicts a single star node representing a first level of systemhierarchy;

FIG. 2 depicts a plurality of interconnected star nodes representing thesecond level of system hierarchy;

FIG. 3 is a diagram depicting representative elements of a typical fullyinterconnected system in accordance with the inventions;

FIG. 3a depicts in greater detail, a portion of FIG. 3,

FIG. 4 is a functional block diagram depicting the principal parts of atypical star node when handling traffic in one direction;

FIG. 5 is a functional block diagram depicting the principal parts of atypical star node when handling traffic in the other direction;

FIG. 6 depicts a typical synchronous multi-user packet;

FIG. 7 depicts a typical synchronous mini-packet suitable for employmentwithin the synchronous multi-user packet of FIG. 6;

FIG. 8 depicts a typical asynchronous mini-packet suitable foremployment within an asynchronous multi-user packet such as that of FIG.12;

FIG. 9 depicts a typical header portion of a synchronous multi-userpacket illustrating a mode indicating no changes in activity fields fora normal synchronous data packet which is transmitted the first time.

FIG. 10 depicts a typical header portion of a synchronous multi-userpacket illustrating a mode for activating activity field addresses;

FIG. 11 depicts another typical header portion of a synchronousmulti-user packet illustrating a mode for changing receive nodes andports for the purpose of operating in circuit switched mode;

FIG. 12 depicts a typical header portion of an asynchronous multi-userpacket;

FIG. 13 is a portion of FIG. 12 in greater detail;

FIG. 14 is a diagram depicting the interrelationships of the satelliteand terrestrial links;

FIG. 15 is a diagram displaying the interrelationships of a satelliteextension node and elements of the network;

FIGS. 16 and 17 are timing diagrams which show time relationshipsbetween star and central nodes.

FIG. 18 is a diagrammatic representation of the contents ofrepresentative speech frames employed by the VADAC-5 vocoders.

FIG. 19 is a diagram providing a comparison of normalized mean queueingtimes; and

FIG. 20 depicts how the instantaneous bandwidth expansion is implementedby joining a mini-packet to a number of extension mini-packets.

Now referring to FIG. 1, it will be observed that there is thereindepicted a central node 1 and a plurality of peripheral (star) nodes2-9. Each of these peripheral nodes e.g., 2, is connected to one otherperipheral (near neighbour) node e.g., 3, via communication paths 10 andto central node 1 over communication paths 11, thus providing analternate transmission path in the event of failure of the principalpath 11a. For example, peripheral node 2 may communicate with centralnode 1 not only via its direct communication link 11a, but additionally,via link 10 to star node 3 and thence through link 11 connecting starnode 3 to central node 1. In such cases, MUP's are switched through node3 unchanged. A "near neighbour" of a peripheral node is anotherperipheral node (not including a central node) that is connected via aone hop communication path.

Those skilled in the art may be accustomed to the use of capitol letters"MP" for the expression multi-user packet and lower case letters "mp"for the expression mini-packet. However in this description theabbreviation MUP is used for multi-user packet in order to distinguishmore particularly from the abbreviation MP which is employed formini-packets.

Now referring to FIG. 2, it will be observed that there are depictedfour central nodes 1', 18, 19 and 20 which themselves are fullyconnected via direct data links 12-17. These data links consist firstlyof 12-14 GHz satellite channels and secondly low capacity fullyconnected terrestrial trunks. The significance of these interconnectionswill become evident in connection with the following detaileddescription of the system operation. It will also be observed that thestar nodes correspond to those of FIG. 1 and one of them is identifiedwith similar symbols for ease of comparison.

In referring to FIG. 2, it will be observed that any node is connectableto any other node through no more than three serial interconnectingpaths. The central nodes are, of course, connected to each other withonly one such path, i.e., paths 12-17. The peripheral (star) nodes, onthe other hand, are interconnectable through either one, two or threelinks, depending upon their physical locations and connections to thecentral nodes. Thus, for example, peripheral (star) node 2' isconnectable to its near neighbour node 3' either via the normal pathconsisting of 11a', node 1 and 11', or in case of emergency, directlyover path 10.

Peripheral nodes connected to different central nodes are connectablevia 3 links. Thus, node 2' is connectable to node 21 via link 11a', link16, and link 22. Further reference to FIG. 2 will demonstrate that inthe absence of link malfunction any node is connectable to any othernode through no more than three communications links. This is asignificant feature of system architecture, for by limiting the possiblenumber of links in transmission paths, it is possible to simplifyinternal communication and to prevent significant delays in both entryto the system and communication therethrough. In this connection, it ishelpful to have in mind that when user information is transmittedthrough the system, the mini-packets maintain their integrity throughoutthe system. However, they are assembled into different multi-userpackets at each subsequent node so that where three hops are involved, amini-packet is included in three different multi-user packets,multi-user packets having a life-time of only one hop.

Now referring to FIG. 3, it will be observed that there is thereindepicted a pair of peripheral nodes 2' and 3' which correspond to likeperipheral nodes in FIG. 2. Also depicted are central nodes 1', and 19again corresponding to like nodes in FIG. 2. Nodes 2' are 3' areconnected to central node 1' via data link 11'. They are alsointerconnected via satellite 23 and communications paths 24-27,synchronous line adaptors 28-31, modems 32-35, transmitter/receiverunits 36-39 operating in TDMA (time division multiple access) mode, andmicrowave antennas (2-3 meter dishes) 40-43.

Each of the nodes, both peripheral and central, are adapted forconnection to lines conducting thereto and therefrom synchronous signalsrepresentative of voice and other data, together with asynchronoussignals of various kinds. Examples of such are depicted in FIG. 3wherein voice signals developed by telephone 44 are connected to vocoder45 where digital signals representative of voice are produced andintroduced through modem 49 to the local loop and then via the modem 46into the synchronous line adaptor 47 to peripheral node 2'. Similarly,asynchronous signals are generated by teletype console 48 and connectedto asynchronous line adaptor 49a (not shown) and modem 50 before beingtransmitted over the local loop 51 and thence through incoming modem 52and asynchronous line adaptor 53 to peripheral node 2'.

Other synchronous data is input from sources such as, for example, acustomer computer 54 or X-25 computer 55 through the front ends 56 and57 and modems 58 and 59 via terrestrial links 60 and 61 and thencethrough modems 62 and 63 and line adaptors 64 and 65 to peripheral node2'.

It may be helpful in understanding FIG. 3 to be aware that theaforedescribed connections to peripheral node 2' are bidirectional. Thatis, not only is data input to node 2' via the above described paths, butreverse data is communicated in the opposite direction therefrom.

Further referring to FIG. 3, provision is shown for connecting to node2' a polled line 66 through asynchronous line adaptor 67 and modem 68.Branching off from line 66 is a plurality of individual asynchronousterminals 67-71 which, for example, may be 1200 baud on-line bankingterminals or 300 baud point of sale terminals in retail stores.

FIG. 3a depicts line 66' and its attached equipment in greater detail.Connected to line 66' are a plurality of on-line banking controllers(e.g. IBM 3601) or the like, connected in full duplex multi-dropmulti-point configuration. As will be recognized by those skilled in theart, 72'-75' are modems. These are, in turn individually connected tothe associated modems 100-107. Modems 100-107 are also connected, asshown, to a number STDM (synchronous time division multiplexing) localor remote simplex loops 108-111 to which there are also connectedpluralities of synchronous modem-terminal pairs such as 112/113,114/115, 116/117 and 118/119. Other terminals (not shown) may beincluded and are represented by the dashed lines such as 120-122.

Further, in connection with FIG. 3a, it will be recognized that whileone loop only is depicted as being associated with each 3601, many otherloops may be advantageously connected thereto. For purposes of clarityin the drawing only one such loop is shown. However, it is contemplatedthat for terminals within the same building that houses the 3601, theremay be a simple simplex digital loop; whereas for remote terminals theremay be one or more additional loops in synchronous time divisionmultiplexing mode, each suitably connected to a set of modems.

Further reference to FIG. 3 reveals that connected to node 19 areequipments similar to those connected to node 2'.

However, such is shown for the purpose of illustration and convenienceonly since each node may have different combinations of equipmentsconnected to it.

It will be evident to one skilled in the art that the variouscommunications links shown in FIG. 3 are adaptable to the type and speedrequired for handling the traffic. Thus, for example, links 11a' and 16will be high speed terrestrial links. (In the drawing, high speed linksare depicted by thicker lines.) The links via satellite 23 similarlywill be high speed. Links 60 and 61 which connect computers 54 and 55into the system may either be a large number of low speed lines or alesser number of higher speed lines in order to transmit the requiredquantities of data into the system. In this connection, as will be morefully developed below, it will be evident to those skilled in the artthat certain types of computers have the capability for development andtransmission of large quantities of data in very short periods of time.Accordingly, very large quantities of data may be transmitted over linkssuch as those identified by symbol 60 and 61 and arrive in star node 2'in batch form, and it is one of the attractive features of this networkthat it is adaptable for receiving and processing such batch data with ahigh degree of rapidity.

Equipment which has been found suitable for use in the herein describedsystem includes VADAC-5 vocoders manufactured by E Systems Corporationof Garland, Tex., U.S.A., standard Bell Telephone System modems, DUP 11and DU 11 line adaptors manufactured by Digital Equipment Corporation ofMaynard, Mass., U.S.A. and for the nodes themselves, PDP 11-45 and PDP11-40 mini-computers also manufactured by Digital Equipment Corporation.Although these equipments are suitable for employment in the system, itwill be evident that many others similarly may be used. Thus, forexample, any suitable linear predictive encoder may be employed forvoice encoding, modems such as those commercially available from theInternational Business Machines Corporation (IBM) may be used with otherIBM equipment, and a variety of mini-computers or microprocessor systemsmay be employed.

Although the foregoing equipment has been used in networks as hereindescribed in connection with FIGS. 3 and 3a, reference to FIGS. 4 and 5may be helpful to a more complete understanding of its operation. Aswill be observed from reference to FIGS. 4 and 5, functional diagramsare therein depicted in which the interrelationships of the variousfunctioning elements of one of the peripheral (star) nodes are shownschematically in block diagram form.

Now referring to FIG. 4 in more detail, it will be observed that thereare shown input and output modems 77 to which incoming and outgoinglines are connected. These modems are in turn connected to line adaptormodules 78'. The upper line adaptor 78' is connected to multi-userpacket and satellite multi-user packet generation and error codingmodule 79 which, in turn, is connected with both the common memory 80and the mini-packet and extension mini-packet generation andmultiplexing module 81. As will be observed, module 81 is interconnectedwith the common memory 80, flow control module 82, instantaneousbandwidth expansion (IBE) module 83 and selective first in/first out(SFIFO) queue handler module 84. Each of the modules 79, 81, 82, 83 and84 are connected to the common memory 80 which in turn is connected tomemory control module 85. In addition, modules 79, 81, 82, 83 and 84 arealso connected to memory control module 85.

The elements of FIG. 5 are similar to those of FIG. 4 except for modules79', 81' and 84' which are different in order to accommodate thedifferent direction of flow. In FIG. 5, module 79' is seen to bedesignated as MUP and SMP packet error checking and disassembly, module81' is MP and EMP packet demultiplexing and sorting, and module 84' isdesignated as outgoing queue handler.

A complete understanding of the operation of FIGS. 4 and 5 can best beachieved from an understanding of the descriptions of the factors whichare involved in the production of mini-packets (MP's) multi-user packetsMUP's) and their transmission through the network. Accordingly,reference should be made to FIGS. 4 and 5 as description of thesefactors is given below. However, briefly, incoming lines are serviced inaccordance with their speed by MP and EMP packet generation andmultiplexing module 81 through module 78 of FIG. 4. There, it isreviewed and if the data is either more or less than that which can beembodied in one MP, the excess is stored in the common memory. At thesame time, the memory control module 85 is kept informed by signalsconducted thereto over the obvious path. Other information received ishandled in a similar manner.

During the next execution cycle of hardware which is many orders ofmagnitude faster than the line service rate, the mini-packets are formedinto an MUP by module 79. The latter also provides appropriate headerand other supervisory signals as well as the cyclic redundancy check(CRC16 or CRC24) to complete the MUP, and the MUP is then transferred tothe line adaptor module for transmission to the next node. Operation ofthe flow control, IBE and SFIFO modules is set forth below.

In FIG. 5, the reverse takes place to that described for FIG. 4. Here,an incoming MUP is checked for errors and disassembled. Afterdisassembly, the mini-packets are transferred to the demultiplexing andsorting module 81', after which the synchronous data is transferred tothe output buffer in the common memory 80 for a period of 40 ms whereasthe asynchronous data is clocked out to the user lines under the controlof modems 77, and queue handler module 84'. In the case of synchronousMUP's, the flow control information is extracted by module 81' and 82and stored in common memory 80.

Ideally, a data and voice communication system would be fullytransparent with respect to both protocol and time. In other words, anideal system would be characterized by having no delays other thanpropagation time therethrough and it would appear to equipment connectedon the far side as if that equipment were connected directly to theterminals of the input side.

All communications systems have some delays due to propagation time. Forexample, in situations where satellite links are used and where thesatellite is approximately 22,000 miles above the surface of the earth,even electromagnetic signals moving at the speed of light will notarrive at their destination until after the passage of more than 250milliseconds. However, in contrast with the network described herein,the networks of the prior art have had delays resulting from one or moreof the following: entrance delays, i.e., time for receipt of enoughcustomer data to form a packet (depends on customer line speed);intermediate link delay, i.e., time needed to receive and error check apacket within the network; intermediate node delay, i.e., time needed toforward a packet within the network (consists of queueing andprocessing); and exit delay, i.e., time needed to transmit a packet ofcustomer data to the destination device.

When considering transparency to user protocol, it is desired thateffective interconnection of user equipment into the system beindependent of the functioning within the user equipment itself; inother words, that the communications system which receives signals fromthe user equipment transmit them rapidly and faithfully through thesystem to the exit port. Thus, transparency in protocol is extremelyimportant since it determines how easily a customer can interface to anetwork. At the host end, this may involve costly and risky changes thatmust be made to the host operating system and telecommunications accesseqipment.

Typical existing equipment has the capability to communicate via leasedor switched lines with terminals using protocols such as BSC or SDLC.Thus, a leased line facility is transparent in both time and protocol.

One well known existing communication capability is that represented byequipment of the International Business Machines Corporation which isbased on full duplex transmission. This is known as SNA. The systemprotocol to support SNA is distributed over the host (i.e., VTAM), frontend (i.e. NCP) and controllers. Interfacing a system to another networkprotocol standard can be very costly (e.g., converting SNA to conform toX-25 international standard protocol for network access). Accordingly,the objective in protocol transparency is to have an intelligent networkfor sharing facilities and providing data sequencing as well as errorcorrection, but which requires a user interface as simple as possible,e.g. EIA RS 232-C, ideally accepting only a serial bit stream consistingof users data and protocol. As hereinabove menioned, the communicationssystem herein described achieves these objectives to a high degree.Accordingly, it will be observed that when the herein described approachis taken, interfacing with the network hereof simplifies tointernational standard RS 232-C, which is designed as an interfacebetween the data terminal and a data set, both at the host computer andthe terminal/controller end.

The network hereof can support any host computer or programmableterminal using X-25 protocol since the first level of this new standardis also RS 232-C. Moreover, the network allows any non-programmableterminal to have network access without requiring any interfaceequipment, and to communicate with its host computer which is programmedto communicate in X-25 only.

The incoming packets entering the network over one of the virtualchannels are stored at the entrance node after implementing the HDLClink protocol. These stored packets will be serviced by formingmini-packets at regular intervals according to their delivery speeds. Atthe destination node, data arriving in the first mini-packet will betransmitted after 40 ms at the same speed behind an X-25 header to thereceiving host.

For users operating in a "switched circuit mode" the call set upprotocol of X-25 will be used in its entirety.

A non-programmble terminal will be serviced as any other low speedterminal entering the network. As the mini-packets reach the destinationnode one by one, they will be stored there until a sufficient number ofbits are accumulated to form an X-25 packet which is then sent to thehost computer over the high speed channel using the HDLC link protocol.

The transparent features of this network eliminate all the interfaceequipment and associated delays. The X-25 users still may enjoy all theadvantages of the network such as transparency in time, flow control,elimination of steady state synchronous queues and freedom from themessage reassembly lockups. Moreover, the low variance of messagetransmission delays will enable the X-25 users to transmit encoded voicemixed with other synchronous data.

The implementation of X-25 in the network system hereof also provides apossible solution to the problem of polling in the network. As will beevident to those skilled in the art, it is expensive for a host computerto perform polling through a shared system using X-25 due to the largeoverhead associated with the small messages. Because of this, networksof the prior art have been forced to use emulation techniques toaccomplish polling and addressing locally. This is very expensive.

An additional attractive feature of the network hereof is that whendealing with non-programmable terminals, X-25 overhead does not gobeyond the receiving node. Hence, the transmission efficiency of pollingis maintained and the use of emulation techniques which make prior artpublic networks dependent on the users' telecommunication access methodsand protocol is avoided.

It will be helpful to an understanding of the following description toview the transportation of data and protocol within the network as storeand forward rhythmic message switching using different multi-userpackets over different terrestrial and/or satellite links.

It will be further helpful to an understanding of the system to considerdata in several categories. First, data which is representative ofvoice; second, synchronous digital data representative of informationdeveloped by high speed intelligent machines, e.g., computers; andthirdly, asynchronous data developed by equipment which is operatedmanually, e.g., teletype machines or on-line terminals. As willhereinafter be described, first priority is accorded data representingvoice; second priority, other synchronous data; thirdly, asynchronousinformation.

Basic operation of the system revolves around periodic development atvery brief intervals of small packets of information hereinafter calledmini-packets (MP's). In the example herein described, such mini-packetsare developed every ten milliseconds for synchronous traffic. However,other brief intervals could readily be employed without departing fromthe principles and scope of the inventive concepts. In this connection,it should be noted that formation of the mini-packets involvesextraction of a predetermined number of information bits from inputbuffers where the bits have been momentarily stored, and consequently,development of the mini-packets is essentially instantaneous. They arethen arranged in sequential order into multi-user packets (MUP's) thelength of which is corrolated with transmission bandwidth so that thenumber of bits of information in an MUP is essentially equal to thenumber that can be transmitted over the transmission links during thefive milliseconds time interval. This formation of mini-packets intomulti-user packets may be considered generally analogous to thesequential placement of box cars in a freight train but where thelocation of each car is known to the system by means of destinationtables (DT's) and activity fields (AF's) which will be described in moredetail hereinafter.

An additional feature of the system includes the provision, wheresubstantial quantities of asynchronous data are to be handled, of MUP'swhich contain asynchronous data as well as MUP's containing allsynchronous data. As mentioned above, data for voice and computercommunications generally is of synchronous type; that is, data isexchanged at regular rates of transmission which are time related.However, with asynchronous data no such time relation is present, forasynchronous data may be received at any random time and, therefore, thepoint in time at which data is received has no particular significance.

In order to handle both synchronous and asynchronous data, provisionsare made for alternate development of synchronous MUP's and asynchronousMUP's in sequence. Thus, for example, a synchronous MUP is followed byan asynchronous MUP which in turn is followed by a synchronous MUP andanother asynchronous MUP. Consequently, each ten millisecond period willsee the formation of one synchronous MUP and one asynchronous MUP, withthe understanding that a synchronous MUP can always preempt anasychronous time slot.

In addition to being used for the transmission of data, MUP's may alsobe used for the transmission of directory or supervisory information.Thus, for example, it is important to the proper operation of the systemthat various nodes have a means of recognizing destination and routingfor each data-containing MP within each MUP. Destination tables (DT's)are developed for this and other purposes.

Destination tables (DT's) are analogous to telephone directories, asthey identify each of the users of the network and the route which theirtraffic should normally follow. Accordingly, the DT at each node for aparticular link will list the following information for its originatingand transient traffic:

Send node

Send port (1-256)

Customer line speed, e.g., 600 baud asynchronous

Traffic priority

Receive node

Receive port (1-256)

The foregoing information is represented by sequential digital signalswhich are encoded in MUP's and transmitted through the network at theinception of operation. Thereafter, they may be changed by subsequentmessages. Consequently, while the DT may be thought of in general termsas a telephone directory, it is subject to rapid and frequent changeduring the course of the day when the system is operating in circuitswitched mode.

In further considering the destination tables, it will be helpful tothink of them as being divided into groups called activity fields(AF's), which indicate the status of the users in DT's i.e. active (1)or inactive (0). Also, in the examples herein described, it will beassumed that links between star nodes and central nodes will each have acapacity of 112 kilobits per second and thus will normally service up to256 terminals or computer ports. For this reason, in the foregoingdescription of the destination tables, reference is made to send andreceive port numbers from 1 to 256.

At star nodes the groups of users identified by Activity Fields one tofive (AF1 to AF5) are serviced, by MUP's containing MP's leaving thestar nodes for the central nodes, in the following sequence of MUPusage: (alternating synchronous and asynchronous)

    __________________________________________________________________________    . . . . 1213121314121312131512131213141213121315121. . . . . . . .            *Activity                                                                            AF2                                                                              AF3                                                                              AF2                                                                              AF3 AF4 AF2                                                                              AF3                                                                              AF2                                                                              AF3                                                                              AF5                                       Field                                                                         *Starting                                                                     at time                                                                       (ms)   5  15 25 35  45  55 65 75 85 95                                        *Packet                                                                       Number                                                                        (Mod 256)                                                                            0   1  2  3   4   5  6  7  8  9                                        __________________________________________________________________________     (*Showing asynchronous only)                                             

    Arrival Rates                                                                          Channel                                                                             Trans-                                                                              Arrival                                                                             Arrival                                                                             Arrival                                               Capacity                                                                            mission                                                                             Rates Rates Rates                                                 ch.'s/                                                                              Rates ch.'s/                                                                              *ch.'s/                                                                             ch.'s/                                       Type of Lines                                                                          sec.  ms/ch.                                                                              100 ms                                                                              25 ms 10 ms                                        __________________________________________________________________________    110 bd async                                                                           10    100   1                                                        150 bd async                                                                           15    66.66 1.5                                                      300 bd async                                                                           30    33.33 3                                                        600 bd async                                                                           60    16.66       1.5                                                1200 bd async                                                                          120   8.33        3                                                  1800 bd async                                                                          180   5.55        4.5                                                1.2 kb/s sync                                                                          150   6.66              1.5                                          2.4 kb/s sync                                                                          300   3.33              3                                            4.8 kb/s sync                                                                          600   1.66              6                                            9.6 kb/s sync                                                                          1200  0.83              12                                           __________________________________________________________________________     (*On the average)                                                        

FIG. 6 depicts a characteristic multi-user packet which, it will beobserved, consists of an initial flag section (F), a HDR (header)section, a mini-packet (MP) section, and an error checking section (BC).

The flag used is an HDLC flag which is a standard international protocoland consists of a zero followed by six 1's and ending with another zero.Its function is to signify the beginning and ending of a multi-userpacket.

The HDR or header section normally consists of two characters, the firstindicating the packet number (useful when a packet contains an error andmust be retransmitted) and the second character being thepriority/format (PF) field. This PF field consists of three sections asfollows:

(i) identification (3 bits)

(ii) service (2 bits)

(iii) extension (3 bits)

The 3 bit identification field is coded as follows:

000: Synchronous MUP--1st transmission--message switched

001: Synchronous MUP--1st transmission--circuit switched

010: Synchronous MUP--retransmission--message switched

011: Synchronous MUP--retransmission--circuit switched

100: Asynchronous MUP--1st transmission--message switched

101: Asynchronous MUP--1st transmission--circuit switched

110: Satellite MUP transmitted over terrestrial links

111: Satellite MUP transmitted using TDMA

TDMA means Time Division Multiple Access

The service field is coded as follows:

00: Loop back

01: Escape

10: Trace

11: Normal data packet

In considering the foregoing, it will be helpful to keep in mind thatnormal type of transmission is store and forward message switching andthat circuit switching occurs only on retransmission or upon linkfailure. In the latter case the life span of an MUP is extended to twohops.

The extension field is coded as follows:

000: There is no change in the activity field and mini-packets will beimmediately following the priority format field.

001: Activate the zeros in the activity field the addresses of which aregiven following the priority format field. (the addresses areconcatenated by 1's and terminated by 0's)

010: Deactivate the 1's in the activity field, the addresses of whichare given following the priority format field. (the addresses areconcatenated by 1's and terminated by 0's)

011: Replace the received node and received port number of the addressin the activity field. (See FIG. 11 in which the address and itsmodifiers are given following the priority format field as showntherein.)

100: Request (or authorization) for "sign on"

101: Communication for "sign off"

110: Request (or grant) for reservation

111: Instruction to cancel reservation

The BC, or block checking section, contains bits which are for checkingin accordance with the CRC16 standard.

FIG. 7 depicts the makeup of a typical synchronous mini-packet (MP). Itincludes from 12 to 96 bits (depending on line speed of theincoming/outgoing lines, and it also includes a trailer section T whichcomprises 3 bits. These trailer bits are:

    ______________________________________                                        Bits     Meaning                                                              ______________________________________                                        111      Not last voice MP, add reservations                                  110      Not last voice MP, cancel reservations                               010      Not last voice MP, no reservation information                        101      Not last data MP, add reservations                                   100      Not last data MP, cancel reservation                                 001      Not last data MP, no reservation information                         000      Last MP, not padded                                                  011      Last MP, padded by 1 bit or more                                     ______________________________________                                    

If the last bit in the data section (just before the three trailer bits)is a 1, then the 011 trailer signifies that only one bit is padded.However, if the last bit in the data section is a zero, the 001 trailersignifies that more than one bit is padded. In this event, all thepadding bits are zeros except the first which is a one.

FIG. 8 depicts the makeup of a typical asynchronous mini-packet. Itincludes a header section H, a character count section CC, and a datasection which may contain up to four characters. The coding for these isas follows:

    ______________________________________                                        Item          Bits     Meaning                                                ______________________________________                                        Header        0        MP has no data section                                               1        MP has a data section                                  Character Counter                                                                           00       1 character                                                          01       2 character                                                          10       3 character                                                          11       4 character                                            ______________________________________                                    

Now referring to FIG. 9, it will be observed that it depicts the flag(as described above) an eight digit packet number (1-256) represented bythe block symbol P# and the priority format field shown as eight digits.The first three digit positions (shown in FIG. 9 as 000) identify themulti-user packet as being synchronous, message switched and the firsttransmission. The next two digits (shown in the figure as 11) signify anormal data packet, and the last three digit positions (shown in thefigure as 000) indicate that there is no change in the activity fieldand the mini-packets will immediately follow.

Another example is shown in FIG. 10 wherein the flag and packet numberidentifications are similar to those of FIG. 9. However, in FIG. 10, thelast three digits of the priority format field (i.e., the extensionfield) is encoded 001. Reference to the above table will reveal that 001indicates that there is to be a change in the activity field and thatthe changes will be given by changing the 0's in the activity field to1's for the addresses which are given following the priority formatfield. Thus, the first address given by eight bits (providing for 256addresses) will be changed from a 0 to a 1, the 0 in the activity fieldlocation representing the second address also will be changed to a 1.The 0 immediately following the block designation of the second addressindicates that address information for the multi-user packet is nowcomplete and that mini-packets will make up the remainder of themulti-user packet.

If instead of 001 in the immediately preceding example, the digits hadbeen 010, this would have signified that the activity field is to bechanged by deactivating the 1's for the addresses which are givenimmediately following the priority format field. As in the previousexample, the addresses are concatenated by 1's and terminated by 0. FIG.11 depicts the initial portion of a multi-user packet having flags andpacket number designations similar to those in FIGS. 9 and 10. However,the last three digits in the priority format field are shown to be 011.Reference to the foregoing table reveals that this signifiesinstructions to replace the received node and received port numbers ofthe address in the activity field. The address and its modifiers aregiven following the priority format field as shown in FIG. 11. The firstaddress consisting of 8 bits is immediately followed by its new node andnew port designators. The new node information consists of 5 bits whichare capable of identifying any one of 32 nodes while the 8 bits withinthe new port designator are capable of signifying any one of theaforementioned 256 ports. The 1 immediately following thereafterindicates that there is to be a second new address together with newnode and new port designations, and the 0 immediately followingthereafter indicates that the new addresses are now completed and thatmini-packets are to follow. This technique is used when a user who isnormally connected through the network to a particular port wants tooperate in "switched circuit" mode. That is, when a user who normally isconnected to a particular port wants on a one time basis to change hisconnection to a different port.

Still considering the coding of the extension field, if instead of the011 digits shown in FIG. 11, they were 110 or 111, the priority formatbyte will not be followed by an address but by 8 bits called a quanta(Q) byte in which 5 bits identify the destination node, 2 bits specifythe number of quanta, and the third bit is used to concatenate Q bytesby 1's and terminate by 0. Q bytes relate to requests for orverification of reservations for traffic through subsequent links. Othertraffic will take precedence over Q bytes and the sending of Q byteswill be performed in the manner hereinbefore described only if there istraffic space to the required destination. If not, then reservationinformation (e.g. add, cancel, command or request reservations) may becarried in the trailer bits of a mini-packet. Another alternative whenthere is a need to send more reservation information than may be fitinto available transmission capacity for an MUP, is to preempt the nextasynchronous MUP and send a packet consisting of the required number ofQ bytes.

Initially, the activity field information is transmitted in itsentirety. After that, as the various terminals sign on and off, theactivity field is modified using the priority format field extension bitsystem described above. Normally, it will be found that on the averageno more than two such address changes are identified in any onemulti-user packet, i.e. during a 10 ms period. Hence on the average, 18bits (2 address bytes +2 extension bits) are required for AF1modifications. Hence when using 112 kb/s links in star nets thesynchronous MUP overhead is approximately 27%, assuming (on average)that an MUP contains 17 MP's of 3 characters each.

When a new address is introduced, a corresponding mini-packet must beincluded in the MUP. At the same time, a 0 may be put in the header ofthe mini-packet which is being discontinued. This is accomplished sincesystem protocol does not allow both activation and deactivation to takeplace in the same packet. In the next packet appropriate deactivationwill occur by including appropriate extension bits following a 010priority format.

Reference is now made to FIG. 12 which depicts asynchronous multi-userpackets (AMUP's). Here, the flag is similar to the flags of the packetshereinbefore described. The first character of the header, i.e. SP#,designates the send packet number which is a number from 1 to 256. Thenext character which is shown as ACK byte is employed for expressacknowledgement or non-acknowledgement of the correct receipt of theprevious eight packets.

It consists of 8 bits which individually signify whether the eight mostrecent packets were correctly received. Thus, for example, in FIG. 13(which shows a part of FIG. 12 in detail), the first bit in the ACK byteis a 1, the next a zero, and next a 1 and so on until the eighth is azero. Beginning with the first bit, the one indicates that the packetwhich is identified by the receive packet number in the RP# byte, hasbeen received correctly. As an example, if that packet is number 79, thenext bit, a zero, relates to the next previous MUP. Since a zeroindicates that the packet has not been received correctly, the zero inthe second position indicates that packet number 78 has not beenreceived correctly. The 1 in the next position indicates that packetnumber 77 has been received correctly, the next 1 that number 76 hasalso been received correctly, the next zero that number 76 has not beenreceived correctly and so on through all eight.

The fourth character in the header is the priority format field. Here,the first three bits and the last three bits of the field have the samefunctions as in the synchronous MUP's. However, the fourth and fifthbits are unique to the asynchronous packets. They identify activityfields as follows:

    00=AF2, 01=AF3, 10=AF4, and 11=AF5

When using 112 kb/s links in star nets the asynchronous MUP overhead is30%, assuming (on average) that an MUP contains 17 MP's of 3 datacharacters each.

It will be recalled that an important feature of the system hereindescribed involves the integration of both satellite and terrestrialcommunications links. Reference again to FIG. 3 will recall theinclusion of "roof top" satellite antennas at the star and central nodelocations. Before proceeding further with an examination of theimportance of these features, it will be helpful to consider that themajor part of real time traffic consists of that which is produced byinquiry response terminals. These terminals characteristically operateat 1200 baud. Moreover, the average message length has been found to beapproximately 50 characters in length. Now considering that thepropagation delay involved in satellite transmission approximates 250milliseconds, it will be observed that for any message of the prior artequal to or longer than 36 characters, the amount of time necessary toreceive such message over conventional terrestrial links will be equalto or greater than satellite propagation time.

Before a packet could be formed in accordance with the prior art, it wasnecessary to receive the message. At the rate of 1200 baud, or 6.66milliseconds per character, the average message length of 50 charactersoccupies approximately 343 milliseconds. Since there are no such delayswithin the system herein described, it will be evident that for suchmessages it will be more rapid to transmit them via satellite than itwould for the prior art equipment to transmit such over directterrestrial links.

An additional feature of the herein described equipment resides in thefact that through satellite communications, each node can be directlyconnected with every other node. Moreover, this can be done with minimumcomplexity and without the necessity for the myriad of interconnectionswhich similar direct connection via land lines would require.

Although satellite links could be employed to accomplish all of theinterconnection functions of the system, a substantial advantage of theinventions herein contemplate the cooperative association of thesatellite links with the terrestrial links whereby increased overallefficiency is achieved. Thus, for example, in order to facilitateretransmission of messages and protocol, substantial savings in time andbuffer storage space can be achieved by providing a limited low speedmirror image of the satellite network on the ground whereby satellitemulti-user packets may be retransmitted for the purpose of errorcorrection, and limited protocol (such as having 6 characters or less)can be speedily sent. To accomplish this, the terrestrial channels wouldoperate at a speed of 2.4 to 9.6 kilobits per second depending upon thetraffic being extended via satellite links.

Another important feature associated with the satellite capabilityinvolves local or remote extension capability. Thus, for example,provision is made for the establishment of a direct entry ability intothe system from any location which may have a volume of trafficsufficient to require it. The attendant advantage is that in such aninstance, a relatively low cost connection via a terrestrial trunk ismade from the extension location into the system, and a large volume oftraffic may be communicated via the satellite to the location to whichthe major portion of the high volume bulk traffic is ordinarilydirected. In such an instance, it will be evident that substantialeconomies will accrue, since a relatively inexpensive terrestrial linewill link the facility with the network, and the major traffic canreadily be handled by the inherently large capacity embodied in thesatellite facility. Further, in accordance with this feature, thepurpose of the terrestrial link centers around the retransmission ofmessages for error correction purposes and the transmission ofappropriate protocol.

Another advantage associated with the integrated satellite/terrestrialnetwork involves its ability to handle error correction of packets whichmay have been damaged either in the case of satellites through weaknessof transmission or atmospherics or in the case of terrestrial linesthrough natural or man-made (impulse) noise. In this connection, it willbe recalled that in the prior art, packets were often formed entirely ofone user's information and consequently any error which may occur withinthat packet would invalidate it in its entirely. Since such a packettypically consists of a number of characters it will be seen that anentire packet of substantial size would require retransmission. However,it has been found that typically, transmission errors in satellite linksresult in clustered errors which occur in no more than a very few of themini-packets within one of the satellite multi-user packets. In such asituation, even if the affected SMUP packet could not be retransmittedwithin the allotted time (before the output buffers would be emptied),and if as a consequence it were necessary to send out to the user theincorrect mini-packets, the user's own error checking of its own messagewould reveal this condition and request retransmission of his messagethrough the network. In such instance it has been found that because ofthe clustered errors, no more than a small number of customers would beinvolved and consequently the entire system would not be burdened with arequest for a large number of message retransmissions. These and otherfeatures of the integrated terrestrial satellite system will be evidentfrom reference to FIGS. 14 and 15 which respectively depict theintegrated terrestrial and satellite subnets and the satellite extensionnode.

Now referring to FIGS. 16 and 17, it is observed that they depict timingrelationships between the star nodes and the central nodes. This may beof particular interest in considering the error correction and checkingfeatures of the system in greater detail. It will be recalled from theforegoing description that certain of the information which istransmitted within multi-user packets relate to the error correcting andretransmission features. Error correction is based on retransmission ofthe multi-user packets between adjacent nodes using an implicit ACK andexplicit NAK scheme. In order to maintain the continuous flow ofsynchronous data and still allow for 3 to 4 retransmissions ofincorrectly received multi-user packets, an output buffer large enoughto hold 4 mini-packets is used at each output port. If during the 40millisecond delay caused by the output buffer, the defective multi-userpacket cannot be correctly received, it is then transmitted forwardbecause the errors are usually confined to no more than 1 or 2mini-packets (customers). A NAK is sent via an HDLC abort character,which preempts the ongoing asynchronous multi-user packet and inserts inthe remaining time slot a null packet. The NAK is received by thetransmitting node before its next synchronous transmission and so itretransmits.

If retransmission is required, then on a central link the multi-userpacket is retransmitted in double redundancy, while on a star link onecopy of the multi-user packet is retransmitted via the alternate (nearneighbour) path in a circuit switched mode and two copies areretransmitted to the central node via the original link. If none of thethree retransmissions is error free, the same retransmission scheme istried once more. The priority format field of the multi-user packetheader indicates mode of retransmission.

In regard to the asynchronous multi-user packets, error correction isbased on retransmission of the multi-user packet. An explicitACK/explicit NAK is used, because the continuity of data transmission isnot so critical for asynchronous multi-user packets. Errorretransmission is initiated by the multi-user packet header, andretransmission occurs in an asynchronous time slot.

As mentioned above, features of the network are very attractive for thetransmission of encoded voice in synchronous MUP's together with otherdata. This is characterized by:

(1) minimal entrance delay (10-20 milliseconds at 2.4 kb/s

(2) low average message delay (70 milliseconds)

(3) low average undetected error rate (10⁻¹² errors/bit)

(4) low variance of message transmission delay.

The minimization of entrance delay is of great concern for encoded voicetransmission. In the network herein described, the entrance delay for a24 bit mini-packet from a 2.4 kb/s encoded voice terminal is 10-20milliseconds. This small delay may be contrasted with that of anon-transparent network such as ARPANET. In such case, direct access tothe network via a TIP involves either the formation of 1000 bit packetswith corresponding entrance delays of over 400 milliseconds for 2.4 kb/sterminals or many smaller packets with very high overhead. The latterapproach reduces the packet formation time but significantly increasesthe variance of message transmission delay and packet sequencingproblems. The high error rate and high delay variance are particularlydamaging to the encoded voice transmission because they result in lossof frame synchronization when statistical multiplexing is employed.

In general, for a synchronous terminal directly interfacing with apacket switched network, the formatting should be such that the networkcan identify the useful information (i.e., busy and idle periods). Inthe case of existing synchronous data terminals with their linkprotocols, these requirements generally are satisfied. However, the sameis not generally true for voice terminals according to the prior art.

In accordance with a feature of this invention, a method of formattingencoded voice is produced which permits direct interfacing with thenetwork hereof by providing a means for identifying active and idleperiods. If the active periods are delineated with HDLC flags, then theleading and trailing flags indicate the start and end of a block ofencoded voice, respectively. HDLC bit insertion is employed to preventthe imitation of flags by the encoded voice data. This accommodatesblocks of variable length, another feature which is important forencoded voice. In this connection, it is helpful to have in mind thatthe average duration of a monosyllabic word and of a talk spurt areabout 1/4 and 1 second, which at 2.4 kb/s represent block lengths of 600and 2400 bits, respectively.

Normally, this protocol will be implemented in hardware such as amicroprocessor as part of the speech processing equipment. Theimplementation is made possible by using the spectrum amplitudeinformation present in each frame of the channel vocoder output.

Suitable vocoders are exemplified by VADAC II and VACAD V speechprocessors and linear predictive coders. Frame structure for 2.4 kb/s(synchronous) operation is shown in FIG. 18 where it will be observedthat frame output of a VADAC V speech processor consists of 54 bitswhich are distributed as follows. First, there are 6 bits representativeof pitch control. These are followed by 1 bit for frame synchronization,and thereafter follow the remaining 47 bits which are spectrumdescriptors.

When the vocoder is idle, all spectrum descriptor bits are low, whereaswhen the vocoder is active, some of the spectrum descriptor bits arehigh. Implementation consists of receive (from VADAC) and transmit (toVADAC) functions.

The receive module performs initial framing and continuous framechecking of the incoming 54 bit frames. Because of byte orientedsynchronous line adaptors (e.g. Digital Equipment Corporation DU11serial interfaces may be used to interface the VADAC's to thehereinbefore mentioned PDP 11's) and the byte oriented bufferingroutines hereof, incoming frames are stored four at a time (27 bytes) tofacilitate byte-by-byte processing.

While an initial sequence of frames is immediately transmitted to enablethe VADAC receiver to acquire initial frame synchronization, this isaccomplished to avoid having the receiver synchronize on the firstactive frame sequence.

Active and idle frames are identified one at a time with no speechclipping and one frame delay (22.5 milliseconds). An alternativeapproach based on predicting the state (active or idle) of the nextframe based on the state of the present one would require no delay butwould result in as much as one frame of speech clipping at the start ofspeech messages.

A hangover is provided at the end of each active frame sequence toensure that the speech message has ended; i.e., stop consonants inspeech are preceded by short pauses (idle periods) that occur in themidst of words and phrases. The hangover extends active periods inspeech to absorb short pauses and thereby form longer continuous speechmessages. This hangover may be variable or fixed depending upon thestatistical properties of stop consonants and the flow control usedwithin the herein described network.

The transmit module performs the following functions. First, the initialframe sequence received is output to the VADAC receiver and is followedimmediately by locally stored idle frames in order to establish andsustain initial frame synchronization in the VADAC receiver. Outgoingidle frames are flagged and frames of newly arrived messages areidentified. Because of the 54 bit vocoder frames and the byte orientedequipment previously described, the first frame pulse in speech messagesis not always located in the first bit position of the first byte, butrather in one of four possible positions. Therefore, a few frames (e.g.5 or 6 ) are examined at the start of each message to identify which ofthe four possible frame sequences is being received. This involves adelay which is absorbed in the network output buffering used forsynchronous error control.

Once newly arrived active frames are identified and outgoing idle framesare flagged, the first of the active frames in each message is meshedwith the last of the idle frames in order to maintain VADAC receiverframe synchronization. Meshing is required because of variable networkdelays which cause the active messages to arrive at randon times withrespect to the locally generated idle frames. Therefore, active framesare buffered until the present idle frame has been output. Normally, avariable meshing delay due to this buffering, of from 0 to 1 frame isincurred. In accordance with the present invention, byte-by-byteprocessing results in meshing delays of from only 0 to 4 frames (90milliseconds); however, these delays which are largely absorbed in thebuffering delay for error control mentioned earlier, and may be reducedto 1 frame maximum through additional processing.

Messages in the network may be classified as type 1, 2, 4, or 8,corresponding to the number of servers required, which in turn dependson user line speed, e.g., 1.2 kb/s messages require one server and 9.6kb/s messages require eight servers.

The terms "bulk" and "job" are used interchangeably, and the term "bulkunit" is introduced.

A "bulk unit" corresponds to 1.2 kb/s, and hence a message of type iconsists of i bulk units.

In considering delays in the network, it may be assumed that there is aninfinite job population, with type 1, 2, 4 and 8 messages and bulkPoisson arrivals, where the bulk size at each arrival instant is random.

As will be recognized by those skilled in the art, a network entrancenode is similar to an M/M/m system, where each of the m servers has 1.2kb/s channel capacity. The non-homogenity of the message populationcauses less than optimal server utilization.

In network entrance nodes, an incoming message is either serviced as itarrives (if sufficient servers are available) or is queued. The networkdoes not service a second message from a terminal while the previousmessage from that terminal is still in service.

If, for example, messages arrive in the following order: type 1, 4, 1,2, 1, 8, 1, 2, 2, 4, if m=number of servers=15; and if a strictly FIFO(first-in, first-out) sequence is used, then one MP from each of thefirst five messages will be packed into an MUP for transmission, leavingfive messages queued and 6 out of 15 servers idle. This is clearly notan effective way to reduce admission delays, and so in the network ofthis invention an SFIFO (selective FIFO) sequence is employed tominimize the number of idle servers. In SFIFO, one MP from each of thefirst 5 messages is packed into the MUP for transmission (as in FIFO),the type 8 message would be queued (first on queue), one MP from each ofthe next 4 messages would also be packed into the MUP, and the type 4message would be queued (second on queue). Hence 2 messages would bequeued, with priority for admission to the next MUP (assuming sufficientservers are available) and only 1 out of 15 servers would be idle.

In accordance with another important feature of the invention,instantaneous bandwidth expansion (IBE) is provided in the network toutilize idle servers to expand the service rate on messages which eitherexperienced admission delay or arrived over a high speed line in batchform. When the IBE technique is not being used, a network entrance nodecorresponds to an M/M/m system. Each time an MUP is formed, themultiple-server system packs the MUP according to an SFIFO discipline.Then, if idle servers remain, IBE will utilize them, but in such afashion as to remain invisible to the multiple-server system. Hence, IBEprovides expanded message service but does not alter either the arrivalstatistics or the average number of servers used by the multiple-serversystem, it remains m.ρ where ρ=utilization factor and m=number ofservers.

IBE may be preempted (by new arrivals) or resumed (if idle servers andcandidates for IBE exist) at each MUP formation time.

The expanded message service provided by IBE shortens the averagemessage length seen by the multiple-server system, and thus shortensmessage service time (since message service time depends to a greatextent on message length at low transmission rates such as 1.2 kb/s or6.66 characters/ms). With IBE, servers are freed sooner than with noIBE, thereby increasing the time transparency of the system by reducingadmission delays. IBE does not affect the arrival statistics ofmessages, which are under user control and assumed to be Poisson. But,by shortening the average message length through the use of differentchannel capacities every 10 ms, IBE convolves a number of differentservice rates such that an M/M/m system without IBE changes to an M/G/msystem. This means that the average message lengths seen by the multipleserver system are now shorter and less random (C_(b) <1), that is, thecoefficient of variation C_(b) is less than one. However, themultiple-server system is not aware of the existence of operation of IBEand thinks that these shorter messages with less random distributionwere generated by the user.

IBE causes messages to be transmitted to the destination node fasterthan they can be output, thereby building a queue at the output node,which permits more retransmissions for error correction than the fewthat are possible with the standard 4 MP output buffer.

Messages which are eligible for IBE have idle servers allocated to themon a FIFO basis. Each allocation represents 12 bits or one quanta ofbandwidth. The portion of the MUP (or the MP in the case of type 1messages), which receives IBE is formed into an extension mini-packet(EMP) consisting of 12 data bits and 1 trailer bit (indicating whetheror not another EMP follows), and is packed into the MUP following the MPfor this message (see FIG. 20).

The protocol embedded in the MP trailer bits indicates whether or notIBE is occurring for this message.

FIFO method for allocation of IBE servers is as follows:

Idle servers, providing they cannot be used by any message in the SFIFO,are allocated to the first message in the FIFO queue (one server foreach 12 bits) until either idle servers or first message are exhausted.If idle servers remain, they are allocated to the second message, untilit is exhausted, etc. This allocation with regard to new arrivals andSFIFO is reviewed every 10 ms.

Not all messages which are candidates for IBE will receive IBE. Some mayescape via normal servicing before IBE can be applied to them. Hence theuse of IBE creates more than one type of message population, and so theavailability of a server depends on what type of message is beingserviced (queued or not queued, expanded by IBE or escaped). Thus thedistribution of admission delays is hyperexponential (C_(b) >1).

The foregoing description of IBE is, by way of example, for single-hoptraffic where there is not a need for reservations. Where multiple hoptraffic is involved, allocation of idle servers for IBE will occur firstfor messages needing no reservations (single hop), second for messagesrequiring and holding extra reservations, and last for messagesrequiring but not holding additional reservations. In other wordsmessages which are undergoing IBE require more reservations and use themfor shorter periods of time since the message lengths are compressed asa result of using IBE.

Unlike a conventional packet switched network of the prior art, thepackets hereof are transmitted and switched in parallel. In other wordseach customer uses a small fraction of a given link capacity, i.e., oneof its channels. Because of this, flow control is called "channel flowcontrol" (CFC).

The motivations for CFC are to:

(i) control congestion in every branch of the network using a globalreservation scheme

(ii) eliminate the synchronous queues at all nodes for the transienttraffic

(iii) reduce the end-to-end message delay variance so that encoded voicecan be transmitted mixed with data

(iv) enable the network to operate with deeply saturated links for longperiods of time

(v) force the queues at the intermediate nodes to approache a D/D/1system representing a completely ordered flow

(vi) prevent the loss of reservations when a node failure takes place

(vii) minimize the admission delays at the entrance nodes by controllingand managing the flow of reservations according to the current trafficrequirements.

Various conventional flow control schemes of the prior art can neitherprovide a global control nor do they allow the branches of the networkto operate in a deeply saturated state. Moreover they fail to providelow enough end-to-end message delay variance so as to enable the networkto transmit encoded voice together with data.

The purpose of channel flow control is to avoid congestion in every linkof the network due to short term traffic peaks and in doing so provideoptimum transit times while allowing maximum channel utilization. In theembodiments herein described flow control applies only to thesynchronous traffic. Moreover, the synchronous traffic always haspriority over asynchronous traffic since the asynchronous user cantolerate reasonable interruptions.

The flow control hereof is based on entities called "reservations". Aparticular reservation is a quantum which corresponds to 12 bits in asynchronous MP travelling in on a particular channel into an exit node(i.e. the last hop). The reservations are required when an MP travelstwo or three links to reach the destination node. When a reservationquantum for a star link is passed from one central node to another node,it also reserves bandwidth on the central link.

Thus an incoming synchronous message must wait for sufficientreservation quanta on the last link to be travelled and then must waitfor available space in an MUP travelling on the first link, in order totransmit the first MP. To reduce this admission delay for thetransmission of synchronous data, asynchronous MUP's can be preempted bythe synchronous MUP's.

The operation of the CFC can be explained in a very simplified manner bythe following queueing model of the airline passengers: Consider thateach node of the star net shown in FIG. 1 is an airport. Also considerthat airplanes (MUP's) of different sizes (channel capacities) carryingpassengers (MP's) are leaving the star nodes at fixed time intervals(10ms) and arriving at the central node at the same time. At the centralnode some of the passengers reach their final destination while theothers catch their connecting flights. CFC forces the queues in thecentral node from D/M/1 towards D/D/1 by simply preventing thepassengers from starting their journeys without obtaining reservationsfor their connecting flights.

The existence of queueing systems approximating D/D/1 at the centralnodes implies that the links of the network can operate using their fullcapacity, i.e. deeply saturated, for indefinite periods of time providedthat the average number of packets entering the network per second anddestined for a particular node is equal to the average message lengthtimes the channel capacity of the saturated link connecting thisdestination node to the nearest central node of the star net.

This is a very definite advantage of the channel flow control and itgoes well above and beyond just providing congestion control.

In a conventional packet switched network, packets are transmitted overhigh speed lines in series. Because of the inherent limitations of anM/M/1 queueing system it is not advisable to operate with the values ofρ higher than 0.7 or 0.8 in a K connected network, if one wishes tomaintain low overflow probability and small buffer sizes. In additionthere are queues in every node of the network. These vary in size andproduce variable delays.

When two or more packets belonging to a particular message are sent totheir destination node using different routes, they experience differentdelays and usually get out of sequence. This end-to-end message delayvariance makes it very difficult to transmit encoded voice and data inthe same network.

If customer packets are transmitted over the same high speed lines inparallel, i.e., each customer occupying a small fraction of thebandwidth, not only will large savings be achieved in storage costs, butalso, idle periods will occur in parallel. This simple means that using"instantaneous bandwidth expansion" (IBE) it is possible to achievetransmission efficiency beyond what is possible with statisticalmultiplexing. Further improvements in transmission efficiency and inend-to-end message delay variance are possible if the high speed linksare divided into 1.2 kb/s channels and each channel is controlled by areservation procedure. Such a procedure is used in the network and iscalled channel flow control (CFC). The best implementation of CFC iswhen using an architecture which consists of a number of star nets. Thecentral nodes can then control and manage the flow of reservations whilestar nets achieve the minimum transit delays by concentrating thetraffic. The central nodes can then be fully connected to each otherusing both the terrestrial links and 12-14 GHz satellite channels. Atone end of the design spectrum one has the choice of selecting theseterrestrial links to be large enough to carry all the real time traffic.While at the other end of the spectrum one can transmit both bulk andreal time data over satellite channels using "roof top antennas", andselecting the terrestrial link channel capacities to be as low as 9.6kb/s. In this case the terrestrial network is only used to transmit theprotocol messages and the retransmission of satellite packets.

The proposed "Channel Flow Control", (CFC) minimizes average synchronousmessage delays in a star net for the full range of P by reducing, if notcompletely eliminating the queues. In other words CFC will force a D/M/1system to approach D/D/1, thereby representing a completely orderedflow.

The existance of queueing systems approximating D/D/1 at the centralnodes makes it possible for the system to operate with fully saturatedbranches i.e., with maximum link efficiency, for long periods of time ifthe channel capacities of these branches are determined by carefultraffic engineering so that the average value of the number of packetsentering the network per second to reach a particular node does notexceed the mean message length times the channel capacity of the branchbetween the destination node mentioned above the nearest central node ofa star net.

The network model given in FIG. 19 shows that the queues can form eitherat the entrance or central nodes. First consider the queues at thecentral nodes. Here, multi-user packets arrive at 5 ms intervals and getserviced. The term servicing entails the following: error checking,sorting out the MP's according to their destination and priorities,forming new multi-user packets and transmitting them at 5 ms intervals.If there are more MP's than can be accommodated in a 66 character packet(2 frames) destined for a particular channel, a queue will startbuilding up at the entrance to that channel.

Clearly the queueing system for each destination is one of deterministicbulk arrivals, bulk service, and a single server. It can be shown thatthe system M/E_(r) /1, in which each customer had to pass through rstages of service to complete his total service, is identical to anM/M/1 system with bulk arrivals of exactly r customers who require onlya single stage of service. Similarly, the system E_(r) /M/1, in whicharrivals taken from an infinite pool of available customers wereconsidered to have passed through r stages of "arrival", is identical toan M/M/1 system which provides service to groups of exactly r customers.

In the example under discussion, the arrivals of multi-user packets atthe central node are not exactly synchronous because the propagationdelay at each channel is different. However, the assumption ofdeterministic bulk arrivals is a good aproximation. With this assumptionthe queueing system can be represented by D/E_(r) /1. Now the queueingtime for the D/E_(r) /1 system falls in the range between D/M/1 andD/D/1 shown in FIG. 19. As can be seen, constant service M/D/1 andconstant arrivals D/M/1 both are about equally effective in improvingthe queue operation, e.g. M/D/1 has half the average waiting time as theM/M/1 system. Also, for large values of line utilization, M/D/1 queuesize approaches 50% of the M/M/1 queue.

On the other hand, in D/D/1 system with regular inputs and constantservice, no queueing can take place. Hence it can be operated up to themaximum utilization ρ=1 with no degradation.

The CFC method can be stated briefly as follows. A synchronous messageto be transported in MP's from a node i to node j, with n≧2, mustreserve a bandwidth equal to its incoming channel capacity on the linkbetween node j and j's central node before it can be allowed to enterthe network.

For messages to be transported between two adjacent nodes, i.e., n=1, noreservations are necessary. The bandwidth of each channel of the starnet will be dynamically allocated to carry

(a) Multi-user Packet and MP overhead.

(b) Adjacent node traffic (n=1).

(c) Transient traffic (n≧2).

The assignments of bandwidth to traffic will be done in quantums of 1.2kb/s, i.e., in 6 bits of 280 bits Data Route frame. Since synchronousmulti-user packets are formed every 10 ms, twice as many bits, i.e., 12bits, will be required for a 1.2 kb/s line. Similarly, 2 quantums ofbandwidth or 24 bits every 10 ms are necessary to support a 2.4 kb/sline.

Although the size of a quantum is fixed at 12 bits, there is a specificquantum for a specific channel of the star net and they are notinterchangeable. The central node controls the movement of the quantumsof reservations to each of its nodes and treats the other central nodesin a similar fashion.

Channel capacity of each network link is calculated to meet the peaktraffic requirement of the estimated user load. Long-term utilization ofchannels in the network is controlled by restrictions on user sign-ons.That is, if a user requests sing-on and has traffic to an exit node forwhich one of the channels along the way is already utilized beyond thepeak traffic limit in the long-term sense, the sign-on request is heldon a "camp-on" queue until enough long-term network capacity becomesavailable (due to sign-off of other users). However in a properlytraffic engineered network the probability of going on a "camp on" queueis very rare indeed.

If the available reservations for a given branch of the network is madeequal to the channel capacity of this branch then there can be nosynchronous queues. However, this is not very satisfactory from thepoint of admission delays. When the system is lightly loaded it isdesirable to have plenty of reservations at the star nodes. This avoidstime delays to request reservations from the central nodes. The channelcapacity is therefore, overbooked as much as 100% and the transientqueues at the central nodes are scanned at regular intervals, e.g.,every 50 ms. If the queue size in front of any branch reaches apredetermined threshold level, then the reservations for this branch arecancelled throughout the network by this particular central node. Whenthe queue clears up, the reservations are once again supplied to thestar nodes, and the system operation goes back to normal.

Reservation information is conveyed by all synchronous MP's travellingfrom a central node to an exit star node, except for the last MP in eachmessage. Hence, for flow control purposes, last MP's are interrupted ascarrying no reservation information. This is consistent with theintention of allowing reservations to remain with the entrance node, tobe used by subsequent messages or to be cancelled later.

Since the central node does all the bookkeeping, if some of thereservations get trapped in a node and cannot be immediately cancelledbecause of lack of suitable MP's, then the central node cancels themusing the extension code 11 of the P/F byte followed up by the Q byte.If a node fails, the central node immediately transfers its reservationsto other nodes. This is strictly a bookkeeping operation.

Clearly, star net is an ideal network topology for the CFC method. Fullyconnected central nodes extend the control across the network. With amaximum propogation delay of 15 ms, the management of reservations isquite practical for the system architecture.

The foregoing examples describe the inventions in a system exmployingcommercial equipment hereinabove identified, said equipment beinggenerally of the high speed byte-oriented type. However, it will beevident to one skilled in the art that other suitable equipment mayreadily be employed. Thus, for example alternate, constructions for themodules depicted in FIGS. 4 and 5 will be evident from the foregoingdescriptions.

The terms and expressions herein are used as terms of description andnot of limitation and there is no intent in the use thereof to excludeequivalents but on the contrary to include all equivalents,modifications and adaptations thereof that fall within the spirit andscope of the invention.

What is claimed is:
 1. An integrated terrestrial/satellitecommunications network having a plurality of peripheral star-connectedand central communications nodes, a first plurality of terrestrialcommunications links connecting said central nodes directly, a secondplurality of terrestrial communications links individually connectingsaid peripheral nodes to said central nodes, a satellite, meansestablishing communications links via said satellite directly betweeneach of said central nodes, means for inputting data-representingnetwork at one node and outputting said signals from said networkthrough another node, means at said one node for selecting certainsignals to be transmitted within the network by terrestrialcommunications links and other signals to be transmitted within thenetwork by satellite links, means at said another node for checking saidother signals for errors and in response to an error in one or more ofsaid signals transmitted via said satellite to transmit back to said onenode via terrestrial communications links exclusively an errorindicating signal, and means at said one node effective upon receipt ofsaid error indicating signals for retransmitting by terrestrial linksexclusively said one or more of said signals.
 2. A network according toclaim 1 wherein said peripheral star-connected nodes are selectivelyconnected terrestrially in near-neighbour pairs.
 3. A network accordingto claim 1 wherein said peripheral star-connected nodes are each furtherconnected by a channel through said satellite.
 4. A network according toclaim 1 wherein said peripheral star-connected nodes are furtherselectively connected terrestrially in near-neighbour pairs and whereeach said peripheral star-connected node is further connected by achannel to said satellite.
 5. A network according to claim 1 whereinsaid outer signals to be transmitted within the network by satelliteinclude signals representing non real time dependent data.
 6. A networkaccording to claim 1 wherein said certain signals to be transmittedwithin the network by terrestrial communications include voicerepresenting signals.
 7. A network according to claim 1 wherein saidcertain signals to be transmitted within the network by terrestrialcommunications include protocol representing signals.
 8. A networkaccording to claim 1 wherein said certain signals to be transmittedwithin the network by terrestrial communications include voicerepresenting signals and protocol representing signals.
 9. A networkaccording to claim 4 wherein said other signals to be transmitted withinthe network by satellite include signals representing non real timedependent data, and wherein said certain signals to be transmittedwithin the network by terrestrial communications include signalsrepresenting voice and protocol.
 10. A network according to claim 4additionally including at least one local or remote entry terminalconnected to an entrance node by a satellite link.